1. Field of the Invention
The present invention relates to an image processing apparatus and an image processing method.
2. Description of the Related Art
There is a process for correcting image deterioration referred to as lens aberration in an image photographed by a camera, using a characteristic value of the lens. Conventionally, distortion and chromatic aberration of magnification among lens aberrations are generated by a deviation between an image height of an image to be formed and the image height which is actually acquired via the lens. The image height indicates a distance from an optical axis. One method for correcting such aberration is to convert coordinates of each of the pixels so that the image height is at a correct position.
In the case of bit map data, i.e., a digital image, the coordinates before and after performing coordinate conversion do not necessarily match. An image transformation process involving coordinate conversion thus generally calculates the pixel values after image transformation is performed as follows. The coordinates after the image is transformed are converted to the coordinates before the image is transformed, and the adjacent pixels are then interpolated according to the converted coordinates.
When such an image transformation process is applied, the edge portion of the image is greatly enlarged, particularly in the case where the process is the lens aberration correction process. Further, the sharpness is greatly reduced when there is a large displacement of the coordinates. A keystone correction process which is performed in a projector is another example of the image transformation process that involves coordinate conversion in which the displacement varies depending on the position of the image.
There are two methods of solving such reduction of sharpness, i.e., by improving the interpolation process and by recovering the reduced resolution.
In the case of improving the interpolation process, it is assumed that the sharpness reduction is caused by the interpolation process. The interpolation process itself is thus recreated by combining the process with edge detection (refer to Japanese Patent Application Laid-Open No. 2004-153668). In such a method, a positive result can be acquired when the image contains very little noise. However, a positive result cannot be acquired in the case of an image which is normally acquired using a digital camera and includes noise.
On the other hand, the reduced sharpness may be recovered by controlling sharpness intensity or controlling the amount of noise.
In the case where the sharpness intensity is controlled to recover the reduced sharpness, the sharpness intensity corresponding to the status of the lens is applied to each pixel to compensate for the reduced sharpness (refer to U.S. Pat. No. 6,603,885). However, the noise included in the photographed image also becomes defocused due to image enlargement, so that a positive result cannot be acquired by only adjusting the sharpness intensity (or a filter radius).
In the case where the amount of noise is controlled to recover the reduced sharpness, noise is added to the interpolation result by considering the amount of noise before image transformation to recover the reduced sharpness due to image enlargement. As a result, the noise reduced by the interpolation process is recovered, and apparent sharpness can be improved. However, if the image to be processed does not contain much noise, the sharpness cannot be improved.
To solve the above-described problems, there is super-resolution processing which is a technique for generating a super-resolution image having a large number of pixels while inputting a plurality of low-resolution images. Since frequency bandwidths of signals included in the low-resolution image are limited, there is a limit in improving the sharpness. The high-resolution image is thus generated using a plurality of low-resolution images. The reduced sharpness due to correcting the lens aberration can be recovered by applying such a technique. For example, Japanese Patent Application Laid-Open No. 2001-197354 discusses a method for acquiring a correction result by determining an evaluation function from a deterioration process of an image pickup system.
There are various super-resolution methods. For example, there is an image capturing method which generates a high-resolution image by combining low-resolution images in which there is a positional deviation between a plurality of frames (Sung C. P, Min K. P, “Super-Resolution Image Reconstruction: A technical Overview”, IEEE Signal Proc. Magazine, Vol. 26, No. 3, P. 21-36, 2003). Further, a method which is based on a maximum a posteriori (MAP) estimation (R. R. Schulz, R. L. Stevenson, “Extraction of high-resolution frames from video sequences”, IEEE Trans. Image Processing, Vol. 5, p. 996-1011, 1996) is well-known. The MAP estimation method is a method for estimating the image in which posterior probability becomes greatest according to a preset probability density function. When the MAP estimation method is applied to the super-resolution processing, it can be seen as estimating a high-resolution image that minimizes the evaluation function in which probability information of the high-resolution image is added to a square error. In other words, when the MAP estimation method is employed in performing the super-resolution processing, the high-resolution image is estimated by solving an optimization problem to maximize the posterior probability by using forecast information of the high-resolution image.
However, the estimation process in the super-resolution processing is generally a very heavy process. For example, in the method discussed in R. R. Schulz, R. L. Stevenson, “Extraction of high-resolution frames from video sequences”, IEEE Trans. Image Processing, Vol. 5, p. 996-1011, 1996, it is necessary to solve the optimization problem to acquire the high resolution image and to perform iterative processing. A large processing load is thus required to acquire an image as a result of the super-resolution processing. If the above-described function is to be realized by a digital single-reflex camera, problems may be generated, such as reduction of continuous shooting speed, or requirement of incorporating a high-speed processor so that power consumption becomes large.